Impaired alertness accompanied by short microsleep events is a frequently reported phenomenon in all areas of modem life. A microsleep event can be defined as a somewhat unexpected short episode of sleep, between 1 and 30 seconds, that occurs in the midst of ongoing wakeful activity. It is suspected that such microsleep events are responsible for many accidents on the road and in the workplace, especially during nighttime. For example, the most notorious industrial accidents of our time, Three Mile Island, Bhopal, Chernobyl, and the Exxon Valdez, all occurred in the middle of the night. Microsleep events can be identified through close inspection of physiological, eye-tracking, video, performance and other alertness-related data. The potentially serious consequences of microsleep events were demonstrated in a study on alertness levels of locomotive operators (see L. Torsvall, T. Akerstedt; "Sleepiness on the job: continuously measured EEG in train drivers"; Electroencephalography and Clinical Neurophysiology 66 (1987), pp.502-511.) During this study, one operator failed twice to respond to a stop signal, because several microsleep events occurred at the time the train passed the signal. The microsleep events were indicated clearly in the electroencephalogram (hereinafter referred to as EEG) and electrooculogram (hereinafter referred to as EOG) recordings.
It is well known in the art that information related to alertness, microsleep events, arousal's, sleep stages and cognition may be discerned from changes in EEG and EOG readings. Unfortunately, not all microsleep events are as easily recognizable as the microsleep events in the aforementioned study of locomotive operators. Often, microsleep events exhibit very complex and diverse characteristics depending on the type of physiological, eye-tracking, video, performance and other alertness-related parameter used for the detection. Furthermore, the characterization of microsleep events is strongly related to the individual person (e.g., EEG type, age, gender, chronotype, etc.) as well as the general alertness level of the person and many other circumstances (e.g., acoustic and optical stimuli, time of day, etc.)
To solve the complex and difficult task of the automatic characterization, detection and classification of microsleep events, a pattern recognition algorithm with several components is needed. These components include for example a data recording system, a feature extraction, normalization and scaling system, an example selection system, an event classification system, an event detection system and a contextual system. A neuro-fuzzy hybrid system (e.g., see C.-T. Lin, C. S. G. Lee; A neuro-fuzzy synergism to intelligent systems, Prentice-Hall, Inc. 1996) would incorporate all the components mentioned above. In addition, neuro-fuzzy hybrid systems are numerical, model-free classifiers, which are able to improve their performance through learning from errors and through their capability to generalize even if they are working in uncertain, noisy, and imprecise environments.
In recent years, a broad variety of neural networks were used successfully for the recognition of many different patterns in physiological data. Neural networks seem to be the perfect tool for the automatic recognition, classification and interpretation of various EEG patterns, such as sleep stages (e.g., see A. N. Mamelak, J. J. Quattrochi, J. A. Hobson; Automatic staging of sleep in cats using neural networks; Electroencephalography and clinical Neurophysiology 79 (1991), PP. 52-61, S. Robert, L. Tarassenko; New method of automated sleep quantification; Medical & Biological Engineering & Computing 30 (1992), pp. 509-517, J. Pardey, S. Roberts, L. Tarassenko, J. Stradling; A new approach to the analysis of human sleep/wakefulness continuum; J. Sleep Res. 5 (1996), pp. 201-210, N. Schaltenbrand, R. Lengelle, J.-P. Macer; Neural network model: Application to automatic analysis of human sleep; Computers and Biomedical Research 26 (1993), pp. 157-171, N. Schaltenbrand, R. Lengelle, M. Toussaint, R. Luthringer, G. Carelli, A. Jacqmin, E. Lainey, A. Muzet, J.-P. Macer; Sleep stage storing using neural network model: Comparison between visual and automatic analysis in normal subjects and patients; Sleep 19 (1996), pp. 26-35, and M. Groezinger, J. Roeschke, B. Kloeppel; Automatic recognition of rapid eye movement (REM) sleep by artificial neural networks; J. Sleep Res. 4 (1995), pp. 86-91), high voltage EEG spike-and-wave patterns e.g., see G. Jando, R. M. Siegel, Z. Hovath, G. Buzsaki; Pattern recognition of the electroencephalogram by artificial neural networks; Electroencephalography and clinical Neurophysiology 86 (1993), pp. 100-109), seizure-related EEG pattern (e.g., see W. R. S. Weber, R. P. Lesser, R. T. Richardson, K. Wilson; An approach to seizure detection using an artificial neural network; Electroencephalography and clinical Neurophysiology 98 (1996), pp.250-272, W. Weng, K. Khorasani; An adaptive structure neural network with application to EEG automatic seizure detection; Neural Networks 9 (1996), pp. 1223-240, and H. Qu, J. Gotman; A patient-specific algorithm for the detection of seizure onset in long-term EEG monitoring: possible use as a warning device; IEEE Transactions on Biomedical Engineering 44 (1997)), micro-arousal (A. J. Gabor, R. R. Leach, F. U. Dowla, Automatic seizure detection using a self-organizing neural network; Electroencephalography and clinical Neurophysiology 99 (1996), pp. 257-266) and for the prediction of Alzheimer disease (e.g., see W. S. Pritchard, D. W. Duke, K. L. Coburn, N. C. Moore, K. A. Tacker, M. W. Jann, R. M. Hostetler; EEG-based, neural-net predictive classification of Alzheimer's disease versus control subjects is augmented by nonlinear EEG measures; Electroencephalography and clinical Neurophysiology 91 (1994), pp.118-130).
The concept of neural networks is very flexible and broad one. Neural networks have been applied to monitor the present somatic state of a human subject (U.S. Pat. No. 5,601,090), to predict the danger of cerebral infarction (U.S. Pat. No. 5,590,665), to detect fear (U.S. Pat. No. 5,568,126), to create a neurocognitive adaptive computer interface based on the user's mental effort (U.S. Pat. No. 5,447,166), to obtain quantitative estimation of blood pressure attributes and similar physiological parameters (U.S. Pat. No. 5,339,818), to establish the difference between a normal and an impaired brain state (U.S. Pat. No. 5,325,862)). None of the neural network prior art discloses the characterization, detection and classification of microsleep events for achieving the goals described herein.
Parallel to the analysis of physiological data using neural networks, a variety of alertness-monitoring systems have been invented. They are based on the determination of alertness through the response to acoustic and optic stimuli (U.S. Pat. Nos. 5,95,488, 5,243,339, 5,012,226, and 4,006,539) or through the correlation between eye and head movement (U.S. Pat. Nos. 5,583,590 and 5,561,693). Recently, a sleep detection and driver alertness apparatus (U.S. Pat. No. 5,689,241) was proposed which monitors and evaluates the temperature distribution in the facial area around the nose and mouth to detect early impending sleep. None of these alertness-monitoring prior arts discloses fuzzy logic, neural networks or any combination thereof. Furthermore sophisticated methods and apparatuses are developed for tracking the eye (U.S. Pat. No. 5.645,550, U.S. Pat. No. 5,620,436), detecting the pupil of the eye (U.S. Pat. No. 5,610,673) and recognizing facial expressions of a person (U.S. Pat. 5,699,449). None of the prior art disclosures in the fields mentioned above are used for the determination if the alertness or fatigue level of a person.
Based on the determination and prediction of alertness and a number of physiological parameters, known in the art as "fatigue countermeasures," e.g., bio-compatible schedules, sleep strategies (U.S. Pat. No. 5,433,223) and methods for modifying a person's circadian cycle (U.S. Pat. No. 5,304,212) have been designed. None of the prior art discloses fatigue countermeasures prior art are designed and validated based on the occurrence, characterization, detection and classification of microsleep events.
In recent publications (e.g., see S. Makeig, T.-P. Jung, Tonic, Phasic, And Transient EEG Correlates Of Auditory Awareness During Drowsiness, Cognitive Brain Research 4 (1996), pp. 15-25; S. Makeig, T.-P. Jung, T. J. Sejnowski, Using Feedforward Neural Networks To Monitor Alertness From Changes In EEG Correlation And Coherence, In: D. Touretzky, M. Mozer, M. Hasselmo (Eds), Advances in Neural Information Processing Systems 8, MIT Press, Cambridge, Miss. (1996); T.-P. Jung, S. Makeig, M. Stensmo, T. J. Sejnowski; Estimation Alertness From EEG Power Spectrum; IEEE Transactions on Biomedical Engineering 44 (1997), pp. 60-69), alertness was estimated by analyzing the correlation between a subject's EEG power spectrum and a local error rate. The error rate was generated from the response of the subject to auditory and visual stimuli. A combination of the EEG power spectrum, a feed-forward multilayer neural network, and principle component analysis was used for the data analysis. This estimation method of measuring and quantifying a subject's level of alertness is not based on occurrence of microsleep episodes. Also, this method of measuring and quantifying a subject's level of alertness is dependent upon the subject being able to take time out from his or her real task to perform the test activities needed to generate a local error rate. Thus, this method and other similar techniques are not suitable when the subject is performing a task requiring his or her full attention, such as driving a vehicle.
It is a special object of the present invention to provide a novel method and apparatus for evaluating and mitigating microsleep events.
It is an object of this invention to provide a method and apparatus for automatic characterization, detection and classification of microsleep events through the simultaneous processing of physiological, eye tracking, video, performance and other alertness-related data obtained from a person while he or she is performing a primary task.
It is another object of this invention to provide a method and apparatus for detecting microsleep events as a function of a subject's physiological measurements.
It is another object of this invention to provide a method and apparatus for detecting microsleep events as a function of a subject's behavioral measurements.
It is yet another object of this invention to provide a method and apparatus for monitoring and quantifying a subject's alertness level without relying upon the subject's response to test activities unrelated to the subject's primary task.
It is a further object of this invention to provide a neuro-fuzzy system that receives a subject's physiological and behavioral measurements and detects the occurrence of microsleep events.
It is a further object of this invention to use examples of fatigue and non-fatigue related events to teach the neuro-fuzzy system.